…action and case staging

Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1
corpus format (#233).

agent-extract.ts turns review.md's '## agent:' sections into data
(name, description, pinned model, prompt body). It takes the markdown
as a string with no fs access, because the baseline arm of an A/B
reads the merge-base review.md via git show. Parsing is strict; a
malformed or model-less section throws listing every problem, since a
silently dropped agent would skew an eval arm without failing it. An
integration test extracts the real review.md (21 agents) and pins the
staging-root reference so a future rename fails a test instead of
silently staging nothing.

live-stage.ts materializes the production staging layout for one
live-enabled corpus case: pr-context.json (review.md Step 1's shape,
synthetic identity fields), full.diff / pr.diff / full-stripped.diff
(all the case diff; corpus diffs carry no generated files and no
pattern-triage pass runs), files.json + review-files.json with the
hasPatch cross-check derived from the diff parse, provenance.json,
routing.json from the deterministic router, an out/ directory, and
the post-change checkout copied from the case tree.
rewriteAgentPrompt swaps the production staging root for the staged
context dir. Everything sits behind an injected-fs seam and is
memfs-tested.

Model dispatch (phase 2c) is deliberately absent; it needs the Agent
SDK dependency decision and arrives separately.

… runner (phase 2c)

live-producer.ts runs the live roster over one staged case behind an
injected LiveAgentRunner seam (the judge.ts pattern), so its logic is
stub-tested with zero model calls. Roster: the default finders
(correctness-reviewer, skill-auditor) plus the router's lensesToSpawn;
no pattern-triage or thread-reconciler in eval. It maps all three
sub-agent output contracts into the shapes the deterministic runner
consumes: label-shape findings (correctness lens, or conventions for
the skill-auditor so labelForFinding reproduces the best-practice
variants; confidence defaulted to 0.7 per the production claims rule),
structured-schema lens findings validated as-is, and the validator's
{claims: [...]} verdicts into CaseVerification[]. It stages
claims.json for the validator, resolves {{#runtime-import}} directives
against the case tree (a case opts into a skills index by carrying the
file), retries once on malformed output with the rejection fed back,
keeps partial results when an agent fails twice, prefixes colliding
finding ids, and reports per-agent cost/turns/wall-clock.

live-runner.ts is the one module that touches a real model runtime:
an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned
to the staged checkout, the agent's pinned model, and hard turn and
wall-clock caps; plus the CLI smoke entry
(tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The
investigation-cap CLI the prompts reference is not staged; its own
denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk
as a dev dependency.

live-match.ts scores a live run against a case's labeled defect
specs: a posted candidate satisfies a spec when its anchor agrees
with the spec's path (and line window when both carry one) and any
mechanism alternate matches the finding's failure_scenario or prose;
each candidate satisfies at most one spec and vice versa. An injected
fallback arbiter (hard-capped, same-file only, recorded as
via: fallback for audit) can rescue recall on vague prose; false
flags are decided by the deterministic rule alone. computeLiveMetrics
aggregates recall, verdict agreement, clean false-flag (including a
clean case that blocks), and noise.

live-ab.ts is the arm orchestrator and CLI: baseline review.md from
git show <merge-base>, candidate from the working tree, both arms
over the same live corpus with everything else (corpus, lib, runner,
metrics, judge) from the candidate, per the plan's settled decision
to isolate the model seam. Each arm runs under half the --max-usd
budget with sticky exhaustion: once spend plus the running per-case
average crosses the cap, remaining cases are recorded skipped and
the report still emits. Spec-level regressions are diffed only over
cases both arms scored. Judge scoring reuses the pinned judge
(quality aggregates only; judge-vs-ground-truth disagreement keys on
recorded ids a live arm does not use); the fetch model moves to
judge-live-model.ts and live-judge.ts now imports it. runner.ts
gains an optional RunOptions.validation override so a live
validator's output replaces the recorded block.

Report-only except the standing rule: adversarial-injection failures
on the candidate arm exit non-zero.

…rpus' into jwbron/live-eval-producer-staging

…staging' into jwbron/live-eval-ab-runner

…s with the case id

Live agents choose their own finding ids, so every case's first
correctness finding was live-correctness-reviewer-1; ids were unique
within a case but collided across cases, and judge.ts's score join
requires arm-global uniqueness. Caught by the first real A/B run
(both arms completed, then judge aggregation threw). Ids are now
<caseId>:<id> from the moment of parsing, so claims.json, the
validator round-trip, the matcher, and the judge all see the same
namespaced id.

…staging' into jwbron/live-eval-ab-runner

…eport instead of killing it

The first real A/B run spent both arms' budgets and then died in
judge aggregation, writing no report: the exact
everything-spent-nothing-posted failure mode the plan forbids. Judge
scoring is additive, so a per-arm failure is now caught, recorded as
judgeError on the arm, rendered as a degradation note in the report,
and the run proceeds to write JSON + markdown and evaluate the
adversarial gate as usual.

…orpus' into tmp-refresh

…roducer-staging' into tmp-refresh

…rpus' into jwbron/live-eval-producer-staging

… the A/B report

The acceptance runs surfaced a claim-validator failure that the
report could only name, not explain (perCase carried agent names
only, and the PerAgentReport.failed detail never reached the
markdown). perCase failedAgents entries are now '<agent>: <reason>',
so the next failure is diagnosable from the sticky comment alone.

…r-row stability footer

The phase 4 acceptance pair measured which report rows a reader may
act on from one run: recall, verdict agreement, the regression lists,
and the adversarial gate reproduced exactly on the no-op control,
while judge quality and noise moved on jitter alone; and on the
weakened arm judge quality went UP 0.16 as recall fell 17 points
(fewer, surer comments each score better). The footer states this on
every report so nobody chases a judge-quality delta or reads one as
health.

…/live-eval-corpus' into jwbron/live-eval-corpus

…gin/jwbron/live-eval-producer-staging' into jwbron/live-eval-producer-staging

…rpus' into jwbron/live-eval-producer-staging

…ron/live-eval-ab-runner' into jwbron/live-eval-ab-runner

…y with backoff)

The judge grades one comment's prose in 512 tokens; every load-bearing
metric (recall, verdict agreement, regressions, adversarial gate) is
deterministic and never touches it, and the acceptance runs showed the
judge signal is not single-run-stable on any model (it moved 0.11 on
the byte-identical control and went UP on the weakened arm). Price
that signal accordingly: the pin moves from claude-opus-4-8 to
claude-haiku-4-5-20251001 (a fifth of the cost; dated snapshot so
week-over-week scores compare; both arms always share one judge, so
within-run deltas are unaffected). Supersedes operator direction 4,
which pinned Opus. Also adds retry with backoff (3 attempts, 429/408/
5xx/network only) to the one live judge seam; a single transient 500
previously wasted an entire weekly scoring pass.

…fetch signature (eslint no-undef)

…-staging

…l-ab-runner